Skip to content

fix variables - #26762

Open
daviszhen wants to merge 19 commits into
matrixorigin:mainfrom
daviszhen:0806-fix-var
Open

fix variables#26762
daviszhen wants to merge 19 commits into
matrixorigin:mainfrom
daviszhen:0806-fix-var

Conversation

@daviszhen

@daviszhen daviszhen commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?

  • API-change
  • BUG
  • Improvement
  • Documentation
  • Feature
  • Test and CI
  • Code Refactoring

Which issue(s) this PR fixes:

issue #25123

#24492

What this PR does / why we need it:

  • 支持未赋值用户变量读取返回 NULL,不再报 “user variable does not exist”。

  • 支持 SELECT ... INTO @var,包括多变量赋值、空结果不覆盖旧值、多行结果报错。

  • 修复用户变量数值表达式:

    • SET @A = 1, @b = 2;
    • SELECT @A + @b;
    • 现在返回 3.0,不再报 TEXT TEXT 类型错误。
  • 修复 prepared statement 参数数值上下文:

    • prepare ps_count from 'select ? + ? as sum_val';
    • execute ps_count using @c1, @c2;
    • 现在可正常返回 3.0。
  • 补充了 planner 单测和 BVT case:

    • pkg/sql/plan/user_variable_numeric_test.go
    • test/distributed/cases/expression/mysql_compat_user_variables.sql
    • test/distributed/cases/expression/mysql_compat_user_variables.result

@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@iamlinjunhong iamlinjunhong left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the complete diff from merge-base 6798bd63884c3fb363589565f925fd16f94eccbe to head 2b290fc5f5c05f3be1105316f42a4c82df23b04b, including parser generation, frontend compile/execute paths, prepared reuse, planner binding, and tests. Requesting changes for two P1 correctness issues; one P2 performance issue is also recorded inline. No P0 or P3 findings.

Comment thread pkg/frontend/status_stmt.go
Comment thread pkg/frontend/select_into_user_variables.go Outdated
Comment thread pkg/frontend/select_into_user_variables.go

@XuPeng-SH XuPeng-SH left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes on exact head 2b290fc5f5c05f3be1105316f42a4c82df23b04b.

I independently traced the parser → planner → frontend/status execution paths and found three blocking correctness gaps:

  1. SELECT ... INTO @vars validates expression/variable cardinality only after receiving a non-empty batch. select 1 where false into @a, @b therefore succeeds and leaves the variables untouched, while MySQL 8.4 rejects it with error 1222 regardless of row count. I reproduced the silent success through MatrixOne's embedded SQL path; the structural check must be independent of runtime result cardinality.
  2. The background execution path installs and fills selectIntoUserVariables, but executeStatusStmtInBack only calls runner.Run and never calls apply. Stored-procedure SQL uses this path, so SELECT ... INTO @var can report success without assigning the variable.
  3. Capture keeps only []any and assignment calls SetUserDefinedVar, which hard-codes IsBin=false. Binary-string metadata from the result vector is therefore lost. Later prepared execution (EXECUTE ... USING @v) consults ResolveVariableIsBin, so values assigned by this new syntax can change type/lookup semantics compared with the existing SET path.

There is also an avoidable unhappy-path cost: the collector detects a second row but does not stop execution, and reports the error only after the entire query has completed. Large inputs continue scanning and transporting rows after the outcome is already known.

Focused parser, planner, frontend, and collector tests pass, but they do not cover these execution/metadata/zero-row boundaries. The zero-row counterexample fails against this head exactly because MatrixOne returns nil error.

@aunjgr aunjgr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the exact head. Requesting changes for three P1 correctness gaps. First, pkg/frontend/select_into_user_variables.go:61 returns before validating expression and variable arity for a zero-row result, so SELECT 1 WHERE FALSE INTO @A,@b silently succeeds. Validate structural arity independently of runtime batches. Second, line 91 stores only the extracted value and loses the source vector binary flag; preserve per-column IsBin metadata and use the binary-aware setter because EXECUTE USING depends on it. Third, the background and stored-procedure path installs the collector but executeStatusStmtInBack never calls apply, so SELECT INTO reports success without assigning variables. Also return the too-many-rows error as soon as rowCount exceeds one instead of scanning the remaining result.

@daviszhen

Copy link
Copy Markdown
Contributor Author

Request changes on exact head 2b290fc5f5c05f3be1105316f42a4c82df23b04b.

I independently traced the parser → planner → frontend/status execution paths and found three blocking correctness gaps:

  1. SELECT ... INTO @vars validates expression/variable cardinality only after receiving a non-empty batch. select 1 where false into @a, @b therefore succeeds and leaves the variables untouched, while MySQL 8.4 rejects it with error 1222 regardless of row count. I reproduced the silent success through MatrixOne's embedded SQL path; the structural check must be independent of runtime result cardinality.
  2. The background execution path installs and fills selectIntoUserVariables, but executeStatusStmtInBack only calls runner.Run and never calls apply. Stored-procedure SQL uses this path, so SELECT ... INTO @var can report success without assigning the variable.
  3. Capture keeps only []any and assignment calls SetUserDefinedVar, which hard-codes IsBin=false. Binary-string metadata from the result vector is therefore lost. Later prepared execution (EXECUTE ... USING @v) consults ResolveVariableIsBin, so values assigned by this new syntax can change type/lookup semantics compared with the existing SET path.

There is also an avoidable unhappy-path cost: the collector detects a second row but does not stop execution, and reports the error only after the entire query has completed. Large inputs continue scanning and transporting rows after the outcome is already known.

Focused parser, planner, frontend, and collector tests pass, but they do not cover these execution/metadata/zero-row boundaries. The zero-row counterexample fails against this head exactly because MatrixOne returns nil error.

  • 每条 SELECT ... INTO 语句重新初始化 collector,修复同一批次多条语句之间状态复用的问题。
  • 保存用户变量赋值时的实际数据类型,数值绑定优先使用变量自身类型,避免被旁边的整数常量错误窄化;支持小数、大整数及数值字符串场景。
  • 扩展 SELECT ... INTO @var 语法,支持 SELECT col INTO @var FROM ... 的 pre-FROM 形式。
  • 零行结果保留原变量值,并增加 MySQL 兼容的 1329 No data warning。
  • 增加多语句、数值类型、prepared statement、pre-FROM、无数据 warning 等 UT/BVT 覆盖。
  • 更新 mysql_compat_user_variables.result,@A + @b 结果由 3.0 修正为 3。

@daviszhen

Copy link
Copy Markdown
Contributor Author

Reviewed the exact head. Requesting changes for three P1 correctness gaps. First, pkg/frontend/select_into_user_variables.go:61 returns before validating expression and variable arity for a zero-row result, so SELECT 1 WHERE FALSE INTO @A,@b silently succeeds. Validate structural arity independently of runtime batches. Second, line 91 stores only the extracted value and loses the source vector binary flag; preserve per-column IsBin metadata and use the binary-aware setter because EXECUTE USING depends on it. Third, the background and stored-procedure path installs the collector but executeStatusStmtInBack never calls apply, so SELECT INTO reports success without assigning variables. Also return the too-many-rows error as soon as rowCount exceeds one instead of scanning the remaining result.

  • 每条 SELECT ... INTO 语句重新初始化 collector,修复同一批次多条语句之间状态复用的问题。
  • 保存用户变量赋值时的实际数据类型,数值绑定优先使用变量自身类型,避免被旁边的整数常量错误窄化;支持小数、大整数及数值字符串场景。
  • 扩展 SELECT ... INTO @var 语法,支持 SELECT col INTO @var FROM ... 的 pre-FROM 形式。
  • 零行结果保留原变量值,并增加 MySQL 兼容的 1329 No data warning。
  • 增加多语句、数值类型、prepared statement、pre-FROM、无数据 warning 等 UT/BVT 覆盖。
  • 更新 mysql_compat_user_variables.result,@A + @b 结果由 3.0 修正为 3。

@iamlinjunhong iamlinjunhong left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the complete diff from merge-base a4b0ce286d182c24efd5a349620ae37016262301 to exact head ce302fc95481492fd2714acb65cdc34251ffc42c, including parser generation, SELECT-INTO normal/background/prepared execution, user-variable type binding and value reconstruction, diagnostics, lifecycle/Q1-Q3 paths, and tests. The author explicitly replied to the previous P2 comments in the PR conversation, and those prior findings are addressed on this head.

This pass confirms three new P1 correctness defects: array-valued user variables can be reconstructed with invalid raw bytes and panic, TIMESTAMP user variables can shift across session/process time zones, and INTO clauses nested in UNION/parenthesized query trees can be silently dropped. No P0, P2, or P3 findings. Requesting changes because P1 blockers remain.

All 26 GitHub checks are terminal with no failures, and git diff --check is clean. A PR-specific targeted-test worktree could not be created because this isolated repository exposes .git/worktrees read-only; no code or worktree files were modified.

Comment thread pkg/sql/util/eval_expr_util.go
Comment thread pkg/sql/plan/base_binder.go
Comment thread pkg/sql/parsers/tree/select.go

@aunjgr aunjgr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed exact head ce302fc after the successful CI rollup. The follow-up closes the earlier blockers: zero-row arity is validated before execution, second rows fail during capture, binary/type metadata is retained, and frontend/background execution both apply the collected variables.

@daviszhen

Copy link
Copy Markdown
Contributor Author

Reviewed the complete diff from merge-base a4b0ce286d182c24efd5a349620ae37016262301 to exact head ce302fc95481492fd2714acb65cdc34251ffc42c, including parser generation, SELECT-INTO normal/background/prepared execution, user-variable type binding and value reconstruction, diagnostics, lifecycle/Q1-Q3 paths, and tests. The author explicitly replied to the previous P2 comments in the PR conversation, and those prior findings are addressed on this head.

This pass confirms three new P1 correctness defects: array-valued user variables can be reconstructed with invalid raw bytes and panic, TIMESTAMP user variables can shift across session/process time zones, and INTO clauses nested in UNION/parenthesized query trees can be silently dropped. No P0, P2, or P3 findings. Requesting changes because P1 blockers remain.

All 26 GitHub checks are terminal with no failures, and git diff --check is clean. A PR-specific targeted-test worktree could not be created because this isolated repository exposes .git/worktrees read-only; no code or worktree files were modified.

  • array/vector 用户变量

    • 用户变量读回时识别 T_array_* / vector 类型。
    • 对真实 Go slice 直接编码成 MatrixOne 内部 array bytes。
    • 避免走 fmt.Sprint(value) 生成 [1 2 3] 这类非法 payload,防止 panic。
    • 覆盖 vecf32/vecf64/vecbf16/vecf16/vecint8/vecuint8。
  • TIMESTAMP 用户变量

    • TIMESTAMP 重建时使用 session timezone,不再依赖进程全局 time.Local。
    • 补充非本地时区和切换 time_zone 后读回的回归测试,避免跨时区漂移。
  • nested / UNION SELECT ... INTO @var

    • AST 层递归提取 SELECT INTO,支持 parenthesized SELECT。
    • UNION 中只允许最后 query block 携带 INTO。
    • 非最后位置的 INTO 明确报 Misplaced INTO clause,不再静默忽略。
    • nested final query block 的 INTO 会产生 MySQL 兼容的 deprecated warning。

@iamlinjunhong iamlinjunhong left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the complete diff from merge-base ff25f27397ab7f1a2952eec98497bfb58419e2c2 to exact head 9ee65d20eca6fefbd91dfb47a1df1a9288d854ea, including grammar/generated parser changes, AST propagation, frontend normal/background/prepared execution, typed user-variable binding/evaluation, diagnostics, and tests. Two P1 blockers remain, plus two P2 compatibility gaps, so I am requesting changes.

Findings are recorded inline:

  • P1: reused array/vector variable executors replace valid element bytes with display-text bytes and can panic on a later batch.
  • P1: INTO clauses in scalar/derived/CTE/EXISTS subqueries are still accepted and silently dropped.
  • P2: mixed pre-FROM/terminal user-variable and OUTFILE clauses can create an AST with both actions, after which export is silently skipped.
  • P2: warning diagnostics are not reflected in the OK warning count, and SHOW ERRORS is no longer filtered to errors.

The previous review on ce302fc95481492fd2714acb65cdc34251ffc42c contained only P1 findings, so the author-response rule for prior P2/P3 findings does not affect this event. Those three prior threads are marked resolved but contain no author replies; the array and nested-INTO follow-ups remain incomplete as described below.

Validation: all 26 GitHub checks are terminal with no failures, git diff --check is clean, and GOWORK=off go test -mod=readonly -count=1 -timeout=120s ./pkg/common/moerr/... passed in the PR-specific worktree. The controlled parser/util/colexec test command could not start because that isolated worktree does not contain cgo/libmo.dylib; I am not treating it as passing evidence.

expr.vec, err = util.GenVectorByVarValueWithAllocation(
proc, expr.typ, val, expr.allocation,
)
} else if !expr.typ.IsVarlen() || expr.typ.Oid == types.T_json {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Re-encode array values when the executor is reused

The first evaluation goes through GenVectorByVarValueWithAllocation and now creates valid element bytes. Every array/vector type is varlen, however, so the second evaluation skips this branch and falls through to SetConstBytes. For []float32{1,2,3}, the default branch stores the seven display bytes [1 2 3]; the next GetArrayAt[float32] panics because the payload length is not a multiple of four. Projection evaluates one executor again for each input batch, so SET @v = CAST(... AS VECF32(3)); SELECT @v FROM a_multi_batch_table can fail on the second batch even when the variable never changes. Treat array OIDs like JSON/fixed values or encode them in the reuse branch, and add a two-evaluation or multi-batch regression for every vector family.

}
return vars, deprecated || node.DeprecatedInto, ""
case *SelectClause:
return node.IntoVars, insideUnion && len(node.IntoVars) > 0, ""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Reject INTO when this Select is nested as a subquery

Every select_stmt reduction calls this top-level helper, including a Select used inside an expression, derived table, CTE, or EXISTS. The nested Select therefore receives IntoVars, but the enclosing top-level traversal only follows Select/ParenSelect/UnionClause; it never visits subquery expressions or FROM/CTE nodes. Queries such as SELECT (SELECT 1 INTO @x) and SELECT * FROM (SELECT 1 INTO @x) AS t are accepted, the outer Select has no IntoVars, the planner has no consumer for the nested field, and the query streams rows while @x stays unchanged. The diagnostic text says INTO is forbidden in subqueries, but no parser-context or complete-AST validation enforces that rule. Please reject these nested forms and cover scalar, derived, CTE, and EXISTS controls.

yylex.Error(intoErr)
return 1
}
if len(intoVars) > 0 && len($6.UserVars) > 0 {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Reject a second INTO regardless of which INTO variant it uses

This only detects user-variable lists on both sides. SELECT a INTO @v FROM t INTO OUTFILE "x" has pre-FROM intoVars but terminal UserVars is empty, so it passes and line 6398 builds a Select with both IntoVars and Ep; executeStatusStmt handles IntoVars first and returns, silently skipping the requested export. The reverse OUTFILE-then-user-variable form also passes. Validate uniqueness across both UserVars and Export, and make export traversal/validation follow the same nested-query rules as variable INTO.

for i := info.length() - 1; i >= 0; i-- {
row := make([]interface{}, 3)
row[0] = "Error"
if i < len(info.levels) && info.levels[i] != "" {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve the protocol contract when adding warning levels

Both SHOW ERRORS and SHOW WARNINGS call this same unfiltered handler, so after a zero-row SELECT INTO adds warning 1329, SHOW ERRORS now returns that Warning row even though MySQL limits it to Error diagnostics. Separately, Session.SetNewResponse still constructs every status response with warnings=0, so the successful SELECT-INTO OK packet does not advertise the warning to connectors/JDBC even though SHOW WARNINGS can find it. Filter by the requested diagnostic statement and pass the current warning count into the response; cover both SHOW variants and the OK-packet warning field.

@XuPeng-SH XuPeng-SH left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep-reviewed exact head 9ee65d20eca6fefbd91dfb47a1df1a9288d854ea across grammar/generated parser, AST ownership, planner binding/cache, frontend normal/background/PERFORM execution, colexec reuse, diagnostics, unhappy paths, and the latest-main merge tree. Four P1 correctness blockers remain:

  1. Array/vector user-variable reuse is still unsafe. The first Eval encodes a typed slice with arrayUserVariableValueToBytes; the reuse path treats every non-JSON varlen type as text and calls SetConstBytes(fmt.Sprintf("%v", value)). I independently reproduced a second-evaluation panic for []float32{1,2,3}: decode slice that is not a multiple of element size. Array-related types need the same typed reconstruction on every value refresh, not the generic varlen text path.

  2. Nested SELECT ... INTO @v still has no execution owner. SelectIntoVariablesForTopLevel traverses only Select/SelectClause/ParenSelect/UnionClause and does not inspect scalar, derived-table, CTE, or EXISTS subqueries. All of SELECT (SELECT 1 INTO @x), SELECT * FROM (SELECT 1 INTO @x) d, WITH d AS (SELECT 1 INTO @x) SELECT * FROM d, and SELECT EXISTS(SELECT 1 INTO @x) are accepted by the parser in a focused counterexample test, but the outer statement has no IntoVars, so assignment is silently dropped. Either reject INTO outside the supported top-level/final query block or propagate it to a single well-defined execution owner.

  3. PERFORM SELECT 1 INTO @x is accepted, but executeStatusStmt handles st.IsPerform before the IntoVars branch and returns after runner finalization without calling the collector apply path. This is another successful silent no-op. Reject this combination consistently with unsupported PERFORM export forms, or define and implement its assignment semantics.

  4. User-variable type binding is stale across the transparent session plan cache. A normal SELECT @v + 0 is cacheable, SET @v = ... is deliberately exempted from ses.cleanCache(), and the cached plan retains the VarRef type resolved from the old assignment. A sequence such as integer assignment → cached select → decimal assignment → same select therefore reuses the old integer expr.typ; runtime reconstruction then parses the current decimal through the stale integer vector type, producing an error or wrong coercion. Ad-hoc SQL must be rebound or the cache entry invalidated when a user-variable assignment can change its type.

Two compatibility gaps are also confirmed and should be closed in this update: mixed pre-FROM user-variable INTO plus terminal OUTFILE is accepted and the frontend silently chooses assignment over export; Warning 1329 is not reflected in the OK-packet warning count, while the shared SHOW handler lets SHOW ERRORS include Warning rows.

Previous blockers for zero-row arity, second-row early failure, binary/type retention, background apply, TIMESTAMP timezone, and final UNION/parenthesized INTO propagation are fixed on this head.

Validation: all 26 GitHub checks are terminal without failure; exact-head full frontend and colexec race tests passed; focused select-into race tests passed at -count=13; standard moerr, frontend, colexec, parser, planner, and sql/util packages passed; git diff --check is clean; and the reviewed tree merges cleanly with the latest local main. Temporary counterexample tests were reverted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Something isn't working size/XXL Denotes a PR that changes 2000+ lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants